Conversation
dpopovTT
requested review from
a team,
djordje-tt and
uaydonat
as code owners
There was a problem hiding this comment.
Pull request overview
This PR updates performance test targets for the Whisper model on Galaxy (wormhole_b0) hardware to account for a regression. The changes acknowledge that the model's performance has degraded, requiring test expectations to be adjusted.
Changes:
- Updated performance metrics for 32-device Galaxy configuration:
prefill_time_to_tokenincreased from 0.21 to 0.22 seconds (~4.8% slower) anddecode_t/s/udecreased from 80.0 to 77.5 tokens/s/user (~3.1% slower)
| 2: {"prefill_time_to_token": 0.13, "decode_t/s/u": 124.0}, | ||
| 8: {"prefill_time_to_token": 0.14, "decode_t/s/u": 105.0}, | ||
| 32: {"prefill_time_to_token": 0.21, "decode_t/s/u": 80.0}, | ||
| 32: {"prefill_time_to_token": 0.22, "decode_t/s/u": 77.5}, |
There was a problem hiding this comment.
The PR description acknowledges a regression in the Whisper model on Galaxy (wormhole_b0), but does not provide an explanation of the root cause. Performance degradations of approximately 4.8% for prefill_time_to_token (0.21 → 0.22) and 3.1% for decode_t/s/u (80.0 → 77.5) should be investigated and documented. Consider adding a comment explaining why this regression is expected/acceptable, or linking to an issue tracking the investigation and potential fix.
### Problem description Currently Whisper model on galaxy regressed, and we need to update the targets ### What's changed Updated targets for `prefill_time_to_token` ### Checklist - [x] [(Galaxy) demo tests - Whisper](https://github.com/tenstorrent/tt-metal/actions/runs/21631324148/job/62345505874)
### Problem description Currently Whisper model on galaxy regressed, and we need to update the targets ### What's changed Updated targets for `prefill_time_to_token` ### Checklist - [x] [(Galaxy) demo tests - Whisper](https://github.com/tenstorrent/tt-metal/actions/runs/21631324148/job/62345505874)
### Problem description Currently Whisper model on galaxy regressed, and we need to update the targets ### What's changed Updated targets for `prefill_time_to_token` ### Checklist - [x] [(Galaxy) demo tests - Whisper](https://github.com/tenstorrent/tt-metal/actions/runs/21631324148/job/62345505874)
2 participants
Problem description
Currently Whisper model on galaxy regressed, and we need to update the targets
What's changed
Updated targets for
prefill_time_to_tokenChecklist